Pictorial digital image processing incorporating image and output device modifications

ABSTRACT

Image-dependent tone and color reproduction processing of digital images is accomplished by creating a spatially blurred and sub-sampled version of the original image, and applying knowledge of the capture device physical characteristics to obtain statistics related to the scene or original captured. The statistics are used for image-dependent linearization of the captured image data based on an OECF model, and in conjunction with information about the output medium, to calculate image-specific preferred tone reproduction curves based on a preferred reproduction model. These curves may be applied to the red, green, and blue channels of RGB-type images to produce preferred tone and color reproduction, or to the luminance channel of luminance-chrominance type images to produce preferred tone reproduction. Alternately, the linearized image data may be reproduced directly. Also, if the original is assumed to already possess preferred reproduction, the preferred reproduction model may be used to undo the preferred reproduction on the original. The image data can then be processed for reproduction on output media with different density ranges and color gamuts. All the processing described can be accomplished automatically, but the access to accurate scene information afforded by the image-dependent linearization, and to the perceptually intuitive parameters controlling the calculation of the preferred reproduction curves, also allows for simple, intuitive manual adjustment.

TECHNICAL FIELD

[0001] This invention relates generally to the processing of digitalimages to produce desired tone and color reproduction characteristics.Specifically, this invention makes use of capture and output deviceinformation, in conjunction with opto-electronic conversion function(OECF) and preferred reproduction models, to determine image-specificprocessing based on statistics derived from the image data. Theprocessing determined may be applied automatically or with user input.

BACKGROUND OF THE INVENTION

[0002] Digital cameras and scanners are used to capture image data froma large variety of scenes and originals. A number of automaticapproaches are employed to process this data for reproduction; but whenreproduction quality is critical, most images are processed manually byexperts. Expertly processed images are also overwhelmingly preferred,even by inexperienced viewers, when comparisons are made.

[0003] Manual processing is time consuming and must be done byindividuals with significant expertise, partly because the controlsfound in currently available software packages make it difficult toachieve desired tone and color reproduction. Simple controls tend tovary the reproduction in ways that miss the optimum, and complexcontrols offer too many degrees of freedom. If a way could be found toproduce results similar to those produced by experts, eitherautomatically or with simple and intuitive manual adjustments, digitalphotography would become a much more attractive alternative toconventional photography.

[0004] The practice of conventional photography suggests thatimprovements in this direction are possible. Currently, conventionalphotographs tend to be superior to automatically processed digitalphotographs in tone and color reproduction, with photographs processedat professional laboratories being far superior. Yet the flexibility ofdigital systems is greater than that of conventional systems. Digitalphotographs have the potential to be better than conventionalphotographs, and expertly processed digital photographs are currently atleast as good. Digital processing approaches that mimic the relativelyfixed behavior of conventional photographic systems should bestraightforward to develop. Insight into digital processing approachescan also be obtained by examining what experts do manually. If this isdone, it is found that most of the decisions made are based onevaluations of the image with respect to the scene or original and withthe desired reproduction goal in mind. It should be possible to developsoftware algorithms that can perform these evaluations and processimages accordingly.

[0005] Three major factors have hindered progress in this area. Thefirst is that expert manual processing is almost always image-dependentand is based on understood tone and color reproduction objectives; butthe development of most digital tone and color reproduction processinghas focused on schemes which do not consider the image data, or considerit without regard for established pictorial considerations. The secondis that the exact meaning of the image data, with respect to the sceneor original, must be known. To date, the approaches used have ignoredmany non-linearities, such as those introduced by optical flare andother image capture effects, and have concentrated on techniques basedalmost exclusively on colorimetry. Colorimetry is strictly applicableonly when the capture spectral sensitivities are color matchingfunctions, or when the colorants used in the original are known andlimited to a number which is not greater than the number of spectralcapture channels. With digital cameras in particular, this is frequentlynot the case. Other difficulties in determining scene physicalcharacteristics have resulted from a lack of standard, accuratemeasurement approaches. When basic flaws are present in a measurementapproach, such as the omission of flare considerations and the fact thatthe spectral characteristics of the detector preclude calorimetricinformation from being obtained, attempts to calculate scene valuesinevitably produce erroneous results. These errors reduce accuracyexpectations and mask other error sources, seriously degrading thecorrelation between captured data and scene characteristics.

[0006] The final factors which have hindered progress are the slowrecognition of the need for preferred reproduction as an alternativegoal to facsimile reproduction, that preferred reproduction is dependenton the scene or original, and that the output and viewing conditioncharacteristics. As mentioned previously, most digital tone and colorreproduction processing development has focused on schemes which do notconsider the image data. Also, color management approaches based oncolorimetry attempt to produce reproductions with colorimetric valuessimilar to those of the original. While calorimetric measures canconsider some viewing condition effects, others are not considered andthe effects of the scene characteristics and media type on preferredreproduction are ignored.

[0007] It is clear that the factors limiting tone and color reproductionquality in digital images stem from an incomplete and sometimesinappropriate global strategy. The inventor has attempted to deal withthese problems in two ways: through participation and leadership in thedevelopment of national and international standards and with theinventions presented here. The following list specifies gaps in thestrategy and the attempts to fill them in:

[0008] 1. Inaccurate and non-standard device measurements.

[0009] Image capture and output devices are measured in a variety ofways, with various measurement and device effects being ignored.Specifically:

[0010] a) Flare and other non-linearities in both capture devices andmeasuring instruments are frequently not considered, or are measured fora particular condition, and the resulting values are erroneously assumedto be applicable to other conditions.

[0011] b) Test targets captured typically have considerably lowerluminance ratios than pictorial scenes, so the extremes of the capturedevice range are truncated or left uncharacterized.

[0012] c) Attempts are made to correlate image data to calorimetricquantities in scenes by capturing data of test targets with deviceswhose channel spectral sensitivities are not color matching functions.Correlations established in this fashion will depend on the test targetused and may not apply to image data from other subjects.

[0013] d) Measurement precision is frequently specified in linear space,resulting in perceptually large errors for darker image areas. Theseareas are also most affected by flare, making dark area measurementsparticularly inaccurate.

[0014] e) Measurement geometries and lighting may be inconsistent orinappropriate to the device use conditions.

[0015] All of these effects compound to produce device characterizationmeasurements which can be quite inaccurate. A common perception of theseinaccuracies is that it is practically impossible to obtain stablemeasurements and, therefore, measurement accuracy need not be too high.Frequently, assumptions are made about the nature of image data, becauseit is felt that the assumption will be as close to the real value as ameasurement.

[0016] However, in conventional photography, standardized densitymeasurement techniques have evolved over decades. These techniquesroutinely produce repeatable measurements with several orders ofmagnitude higher accuracy than those obtained for digital systems, whichis one of the reasons the less flexible conventional systems are able tooutperform current automatic digital systems. Unfortunately, the reasonthese techniques are so accurate is because they have been refinedspecifically for conventional photographic materials. A great deal ofwork will be required to develop similar techniques for the devices andmaterial used in digital systems.

[0017] Work has just begun in this area, but significant progress isalready being made. A new standard to be issued by the InternationalOrganization for Standardization (ISO), “ISO 14524,Photography—Electronic still picture cameras—Methods for measuringopto-electronic conversion functions (OECFs),” is almost complete, andwork has begun to develop two new standards: “Digital still picturecameras—Methods for the transformation of sensor data into standardcolour spaces” and “Photography—Film and print scanners—Methods formeasuring the OECF, SFR, and NPS characteristics.” While these effortsare collaborative and open, the inventor is the project leader for theformer two efforts, and participates in the latter.

[0018] 2. Difficulties in communicating device information due to a lackof standard data types, terms, and data formats.

[0019] Even if accurate measurements are available, a completeprocessing strategy requires that the measurements characterize thedevice in question by completely filling with values an enumerated listof expected measurements. These values must also be provided in theimage file in a standard format to be readily usable by a variety ofprocessing software. “ISO 12234/1, Photography—Electronic still picturecameras—Removable memory, Part 1: Basic removable memory referencemodel” and “ISO 12234/2, Photography—Electronic still picturecameras—Removable memory, Part 2: Image Data Format—TIFF/EP” define thecharacterization data required and where and how it is to be included inimage files. The inventor was a key participant in the development ofthis standard, particularly the enumeration of the characterizationdata. It should be noted that because of a lack of consensus concerningthe need for some of the data among the members of the ISO workinggroup, some of the data categories are defined as “optional” at present,but the necessary categories are described.

[0020] 3. How to deal with image-dependent capture non-linearities.

[0021] The measurement methods described in the standards mentionedabove tell how to measure digital photography system characteristicsusing various test targets, but they do not deal with methods forestimating image-dependent capture non-linearities. A solution to thisproblem is described in this patent application.

[0022] 4. Lack of specification of standard, optimal methods fortransforming image data from the capture device spectral space tostandard color spaces.

[0023] A number of methodologies have evolved for transforming capturedevice data into intermediate or standard color spaces. Many of thesemethods have merit in particular circumstances, but in many cases areapplied inappropriately. The lack of accurate characterization datacompounds the problem in that it is difficult to tell if the cause oflow quality transformed data is an inappropriate transformation method,inaccurate characterization data, or both.

[0024] Another difficulty has been that, until recently, the onlystandard color spaces used for digital photography were those defined bythe CIE (Commission Internationale de L'Éclairage or InternationalCommission on Illumination) based on the human visual system (HVS). Forvarious reasons, it is generally impractical to design digitalphotography systems that mimic the HVS. Most digital photography systemsanalyze red, green, and blue (RGB) light; and most output devicesmodulate these spectral bands. In conventional photography, specific RGBbands are well defined by the spectral characteristics of thesensitizing dyes and colorant dyes used and by standards such as “ISO7589, Photography—Illuminants for Sensitometry—Specifications forDaylight and Incandescent Tungsten” (which defines typical film spectralsensitivities) and “ISO 5/3, Photography—Density measurements—Spectralconditions.” These standards were developed many years ago, but theinventor actively participates in their maintenance and revision.

[0025] In digital photography, a wide variety of spectral sensitivitiesand colorants are used by different systems. Many of these systems arebased on RGB analysis and synthesis, but the data produced in capturinga particular scene can vary substantially between capture devices. Also,if the same RGB image data is provided to different output devices,significantly different results will be obtained; and the differenceswill not be, for the most part, the results of optimization of the imagedata to the media characteristics.

[0026] Over the past five years, companies involved with digital imaginghave recognized this problem and invested significant resources insolving it. Some progress has been made, particularly within theInternational Color Consortium (ICC), an association comprising most ofthe major computer and imaging manufacturers. However, the efforts ofthe ICC have been directed at producing consistent output from imagedata. The metrics employed are based on colorimetry and generally aim toproduce output on different devices that is perceptually identical whenviewed under a standard viewing condition. This aim is commonly referredto as “device-independent color.” Device-independent color is anappropriate goal in some cases, but frequently falls short. Differentmedia have vastly different density range and color gamut capabilities,and the only way to make sure that all colors are rendered identicallyon all media is to limit the colors used to those of the lowest dynamicrange (density range and color gamut) medium. This is certainly notdesirable, and consequently a number of ICC member (and other) companiesare now creating “ICC profiles” that produce colors from the same imagedata which vary between devices. (ICC profiles are device-specifictransformations in a standard form that ostensibly attempt to transformimage data to produce device-independent results.)

[0027] The basis for the color science on which device-independent coloris based is the behavior of the HVS. Much of this behavior is reasonablywell understood, but some is not, particularly the effects of some typesof changes in viewing conditions, and localized adaptation. Also, theeffects of media dynamic range on preferred reproduction have little todo with the HVS. Appearance models may be able to predict how somethingwill look under certain conditions, but they give no information aboutobserver preferences for tone and color reproduction in a particularphotograph.

[0028] ICC profiles are currently being produced that attempt totransform captured image data to produce colorimetric values (inputprofiles), and that take image data and the associated input profile andattempt to transform the colorimetric values to data suitable for outputon a particular device (output profiles). These profiles are generallyconsidered to be device-specific, in that a particular input profile isassociated with a particular capture device and a particular outputprofile is associated with a particular output device. This type ofassociation makes sense in view of the philosophy of the approach. If ascene or original is characterized by particular colorimetric values,the goal of the input profile is to obtain these values from thecaptured data. Likewise, the goal of the output profile is to reproducethe calorimetric values on the output medium. Since the goal isfacsimile calorimetric reproduction, the profile should be independentof the scene content or media characteristics.

[0029] If the capture device spectral sensitivities and/or colorantsused in the original make it possible to determine colorimetric valuesfrom captured data, it is theoretically possible for ICC-type inputprofiles to specify the appropriate transformations. Also, if thecharacteristics of the capture device do not vary with the scene ororiginal captured and the device spectral sensitivities are colormatching functions, a single profile will characterize the device forall scenes or originals. If knowledge of the colorants is required toallow calorimetric data to be obtained, a single profile is adequate foreach set of colorants. Unfortunately, flare is present in all capturedevices that form an image of the scene or original with a lens (asopposed to contact type input devices, like drum scanners). The amountof flare captured will vary depending on the characteristics of thescene or original. Occasionally, other image-dependent non-linearitiesare also significant. For ICC profiles to specify accuratetransformations for devices where flare is significant, not only mustthe calorimetric spectral conditions be met, but the image-dependentvariability must be modeled and considered. The resulting input profilesare dependent on the distribution of radiances in the scene or original,as well as the capture device used and the colorants (if applicable).

[0030] In summary, the primary difficulties with using ICC profiles tospecify transformations are:

[0031] a) ICC input profiles only allow transformations to CIE colorspaces, yet transformations to this type of color space are valid onlyif the capture device sensitivities are color matching functions, orcolorants found in the scene or original are known, and are spanned byspectral basis functions not greater in number than the number of devicespectral capture channels. These conditions are almost never met whendigital cameras are used to capture natural scenes.

[0032] b) The appropriate ICC input profile for a particular deviceand/or set of colorants to be captured is generally assumed to beinvariant with the content of the scene or original. This assumption isnot valid with the many capture devices that have significant amounts offlare, such as digital cameras and area array scanners.

[0033] c) The measurement techniques used to determine ICC profiles arevariable, and the profiles constructed are usually not optimal.Frequently, profiles are not linearized correctly, neutrals are notpreserved, and incorrect assumptions are made about the colorants in thescene or original. These inaccuracies are masked by and compounded withthe inaccuracies mentioned in a and b.

[0034] d) While it is recognized that different output media mustproduce different calorimetric renderings of the same image data for theresults to be acceptable, there is no standard methodology fordetermining how to render images based on the dynamic range of the sceneor original, as compared to the output medium.

[0035] The ICC efforts have resulted in a significant improvement overdoing nothing to manage colors, but in their current manifestation arenot viewed as a substitute for manual processing. Fortunately, the ICCapproach is continuing to evolve, and other organizations are alsocontributing. In particular, there is a proposal in the ICC to allowanother standard color space based on a standard monitor. This colorspace is an RGB space, making it more appropriate for use with manycapture devices, particularly RGB-type digital cameras and filmscanners. This proposal is also being developed into a standard:“CGATS/ANSI IT8.7/4, Graphic technology—Three Component Color DataDefinitions.” The inventor is involved in the development of thisstandard. Also, the proposed new ISO Work item mentioned previously, forwhich the inventor is the project leader, “Digital still picturecameras—Methods for the transformation of sensor data into standardcolour spaces,” is specifically aimed at specifying methods fordetermining optimal transformations. As these efforts are completed andif the methods for dealing with image-dependent non-linearities of myinvention are used, it should become possible to specify capture devicetransformations that are determined in standard ways and based onaccurate measurements.

[0036] 5. How to determine preferred reproduction based on thecharacteristics of the scene or original and the intended output medium.

[0037] The first part of the digital image processing pipelinetransforms capture device data into a standard color space. Once thedata is in such a color space, it is necessary to determine an outputtransformation that will produce preferred reproduction on the outputmedium of choice. A method for accomplishing this is contemplated by myinvention.

[0038] 6. How to take image data processed for preferred reproduction onone output medium and transform it for preferred reproduction on anotheroutput medium.

[0039] Frequently, it is necessary to take image data which has alreadybeen processed for preferred reproduction on one output device andprocess it for preferred reproduction on another output device. A methodfor accomplishing this also contemplated by my invention.

[0040] 7. How to implement user adjustments that produce preferredreproduction with maximum simplicity and intuitiveness.

[0041] As stated previously, current manual processing software tends tobe overly complicated and offers too many degrees of freedom or isincapable of producing optimal results. The implementation of thepreferred reproduction model described in this patent application allowsfor user specification of key parameters. These parameters are limitedin number; and changing them produces transformations which alwaysproduce another preferred rendering, limiting the possible outputs tothose that are likely to be preferred.

SUMMARY OF THE INVENTION

[0042] Embodiments of my invention, in conjunction with theabove-mentioned international standards under development, solve theabove-identified problems by providing a complete strategy for theprocessing of digital image data to produce desired tone and colorreproduction characteristics. The details of this strategy are asfollows:

[0043] 1. A scaled version of the image is constructed by spatiallyblurring and sub-sampling each channel of the image data. The scaledversion is preferably a reduced version, but can be of any scale withrespect to the original image. The blurring and sub-sampling areaccomplished using one or more filters that first blur the image datausing a blur filter with a radius that is primarily related to thenumber of pixels, rows of pixels, or columns of pixels in the imagechannel, but can also be affected by other factors, such as the intendedoutput size or pixel pitch, the intended output medium, the numericalrange of the image data, etc. Any common blur filter can be used, suchas a boxcar averaging or median, a Gaussian blur, etc. The blurred imageis then decimated to produce the scaled image, which is stored forfuture use.

[0044] 2. The capture device focal plane OECFs are determined for eachchannel according to ISO 14524 for digital cameras or the standard whichresults from the new work item under development for scanners. Theinverses of these OECFs are then determined, either in functional formor as look-up-tables (LUTs). This information may also be provided bythe device manufacturer or included in the image file header with somefile formats.

[0045] 3. The scaled image data is transformed into focal plane datausing the inverse focal plane OECFs. Statistical values are thendetermined for each channel from the transformed data. Typicalstatistical values are the minimum and maximum focal plane exposures,the mean focal plane exposure, and the geometric mean focal planeexposure. Other statistical values may be determined in some cases.

[0046] 4. The capture device design and OECFs are evaluated to determineif the capture device has significant image-dependent non-linearities orflare. If image-dependent effects are found, they are modeled. The modelto be produced should predict the amounts of non-linearities and flarebased on statistical values determined from the scaled image data.Models can be constructed by capturing a variety of scenes or originals(such as ISO camera OECF charts with a variety of luminance ranges andbackground luminances), determining the flare and non-linearitiesencountered when capturing these charts, and then correlating themeasured values with the scaled image statistics. Flare models can alsobe constructed by compounding extended point-spread-functions. A flaremodel may be provided by the device manufacturer, but there is nomechanism at present for including this information in the file format.

[0047] 5. The estimated camera or scanner OECFs for the imagerepresented by the scaled image are determined for each channel usingthe OECF measurement standards mentioned, in conjunction with the flareand non-linearity model. The inverses of these OECFs are thendetermined, either in functional form or as LUTs. These inverse OECFs,which will be referred to as the input linearization information orinput linearization tables, are stored for future use.

[0048] 6. The capture device spectral sensitivities are evaluated and anappropriate transformation to an intermediate spectral or color space isdetermined. This intermediate color space is preferably a color spaceappropriate for application of the preferred reproduction model, such asa standard color space. If the intermediate color space is a standardcolor space, the transformation can be determined according to one ofthe methods in the proposed new ISO standard. In this case, the inputlinearization table is used to linearize the captured data, as requiredby the standard. The transformation may also be provided by the devicemanufacturer or included in the image file header with some fileformats.

[0049] 7. The scaled image is linearized using the input linearizationtable and transformed to the intermediate color space using thetransformation determined. A luminance channel image is then determinedusing the equation appropriate for the intermediate color space.Statistical values are then determined from the luminance channel data.Typical statistical values are the minimum and maximum (extrema) sceneluminances, the mean luminance, and the geometric mean luminance. Otherstatistical values may be determined in some cases. The scaled imagedata is generally not needed after these statistical values aredetermined.

[0050] 8. The output device is determined, either by assuming it to be astandard monitor, by asking the user, or by the software (if intendedfor a specific device). The visual density ranges of all selectableoutput devices should be known. The viewing conditions under which theoutput will be viewed may also be specified.

[0051] 9. The statistical values determined from the luminance channelof the scaled image, the density range of the output device, and theviewing illumination level (if known) are input to the preferredreproduction model. This model calculates an image and output specificpreferred tone reproduction curve. This tone reproduction curve istypically applied to RGB channels, to produce preferred tone and colorreproduction.

[0052] 10. The output device electro-optical conversion function (EOCF)characteristics are determined by measuring the output of the device forall possible input digital levels or, in the case of the standardmonitor, by using the standard monitor EOCF. An output transformation isthen determined by combining the preferred tone reproduction curve withthe output device EOCF. This transformation may be expressed infunctional form or as a LUT and will be referred to as the output table.

[0053] 11. The image data for the entire image is linearized using theinput linearization tables. It is then preferably transformed into theintermediate color space. This color space can be a standard RGB space,although monochrome image data should be transformed into aluminance-type space, and this processing also may be used to producedesired tone reproduction characteristics with luminance-chrominancetype color space data. The output tables are then applied to the linearintermediate color space data to produce digital code values appropriatefor a standard monitor or the specified output device. If necessary,standard RGB values or other color space values corresponding topreferred reproduction may be converted to another color space for useby the output device. In this case, the goal of the processing employedby the output device is to produce a facsimile reproduction of thepreferred reproduction as expressed in the standard RGB or otheroriginal color space. The preferred reproduction should have beendetermined with consideration of the luminance range capabilities of theoutput medium.

DESCRIPTION OF THE DRAWINGS

[0054]FIG. 1 is a schematic flow chart of the overall method of theinvention.

[0055]FIG. 2 is a schematic flow chart of the step of gatheringpreliminary information illustrated in FIG. 1.

[0056]FIG. 3 is a schematic flow chart of the step of providing anoriginal image illustrated in FIG. 1.

[0057]FIG. 4 is a schematic flow chart of the step of constructing ascaled image from or a scaled version of the original image illustratedin FIG. 1.

[0058]FIG. 5 is a schematic flow chart of the step of analyzing thescaled image or scaled version illustrated in FIG. 1.

[0059]FIG. 6 is a schematic flow chart of the step of processing theoriginal image to produce a processed or output image illustrated inFIG. 1.

[0060]FIG. 7 is a schematic flow chart of another embodiment of theinvention.

[0061]FIG. 8 is a schematic flow chart of another embodiment of theinvention.

[0062]FIG. 9 is a schematic flow chart of another embodiment of theinvention.

[0063]FIG. 10 is a schematic flow chart expaning on the embodiment shownin FIG. 9.

[0064]FIG. 11 is a schematic flow chart expaning on the embodiment shownin FIG. 9.

[0065]FIG. 12 is a schematic flow chart of another embodiment of theinvention.

[0066]FIG. 13 is a schematic flow chart expaning on the embodiment shownin FIG. 12.

[0067]FIG. 14 is a schematic flow chart expaning on the embodiment shownin FIG. 12.

[0068]FIG. 15 is a schematic flow chart of another embodiment of theinvention.

[0069]FIG. 16 is a schematic flow chart of another embodiment of theinvention.

[0070]FIG. 17 is a schematic flow chart of another embodiment of theinvention.

DESCRIPTION OF THE INVENTION

[0071] The processing strategy described herein and shown schematicallyin the accompanying Figures and Tables seeks to accomplish four goalssimultaneously:

[0072] Process image data to produce the best possible result in termsof what is desired by the user.

[0073] Minimize complexity whenever possible in order to reducecomputational requirements and emphasize the basic function of theprocessing algorithms employed.

[0074] Automate the processing to the greatest extent that is consistentwith hardware capabilities and user quality expectations.

[0075] Improve the efficiency of user adjustments by focusingcapabilities on the more likely outcomes and making the adjustmentprocess as intuitive as possible.

[0076] The above goals force processing strategies in specificdirections. In particular, it is desirable to consider the physics ofimaging systems. Many operations are best performed with the image datain a particular physical representation. Also, physical measurements ofthe behavior of components in each system can be extremely useful indetermining processing parameters. Since little of this information isobtained by the user, it is desirable to automate the transfer of thisinformation, either as part of the image file or between devices and theprocessing software. Several newer image file formats accommodate thistransfer.

[0077] Another consideration in the development of the processingstrategy is device-independent performance optimization. Digital imagedata comes from a variety of sources and may be used for a variety ofpurposes. For any strategy to be truly useful, it must be able toproduce excellent results on a large variety of devices.Device-independent performance optimization, however, should not beconfused with most current manifestations of device-independent color.Optimized performance occasionally results from reproducing calorimetricmeasurements; but frequently an optimized reproduction will be somewhatdifferent from the original, particularly with photographs. Some ofthese differences are attributable to differences in human visual systemadaptation. Development of a truly comprehensive appearance model andthe reproduction of appearance would undoubtedly produce optimizedreproductions in many cases. Such a model would by necessity beextremely complex, however, and might require data about surround andviewing conditions, etc., that are difficult to obtain. Also, in somecases, even the reproduction of appearance might not produce thepreferred result.

[0078] For many decades, photography has evolved an empirical set ofpreferred reproduction goals. These goals have been driven to someextent by materials considerations, but the relatively high qualityceiling of the photographic process prevents media limitations fromgreatly affecting the goals. A more significant problem is that thegoals were not extensively documented. Also, the relative rigidity ofchemical processes prevents the goals from being tweaked in ways thatwould be advantageous with more flexible systems, such as digitalsystems. Nevertheless, the implementation of these goals in conventionalphotographic systems has resulted in pictorial imaging systems capableof producing preferred reproductions of excellent quality and relativeinsensitivity to changes in the viewing environment.

[0079] Recently, attempts have been made by the inventor to documentpreferred photographic reproduction goals and extend them forapplication to digital processing. A result of this work is theemergence of several issues commonly considered to be of majorimportance in photography. In particular, the effects of flare, imagekey (high- or low-), scene dynamic range, viewing conditions, andveiling glare are addressed. Addressing these issues fordevice-independent performance optimization in digital photographyrequires that the proposed processing strategy be scene and outputviewing condition dependent. Photography deals with output viewingconditions through the use of specific media and standardrecommendations for specific applications. Scene dependent issues aredealt with by engineering materials for graceful failure and by humanintervention at both the image capture and processing stages. Withdigital systems, it is possible to shift the scene dependentintervention to smart processing algorithms.

[0080] Tone and color processing are of major importance in producingexcellent images, but spatial processing can also have a significanteffect on quality. The expense of manufacturing one-shot digital cameraswith adequate numbers of pixels and the use of color filter arrays hasincreased the importance of spatial issues even further. Over the pastdecade, research in spatial processing has been largely separate fromcolor processing, but device-independent performance optimizationrequires that the two be integrated. Optimized spatial processing isalso becoming more output dependent. The spatial frequency capabilitiesof the output media and the viewing distance affect the degree ofsharpening desired. Photographic artists and engineers have long knownthat mean-square-error (Wiener filter) based restoration is a starttoward optimized processing, but that additional edge enhancement isneeded. Recent work in information throughput based restoration isbeginning to suggest new mathematical approaches to spatial processing.

[0081] In discussing processing strategies, it is important todifferentiate between pictorial processing to produce specificreproduction goals, image editing, and image manipulation. Image editingimplies direct manual alteration of the image data to produce somedesired result. With image manipulation, all or part of an image isintentionally altered to produce a specific effect or make some point.Moving objects around, changing peoples' faces, radically changing thecolors of objects, and distortions are image manipulation. Taking animage and processing it to produce a pleasing result is pictorialprocessing. The boundary between the two can blur when considering howmuch of an increase in contrast or saturation is preferred, as opposedto exaggerated. Most popular photographic image processing applicationsare well suited for image editing and manipulation. The processingstrategy outlined here is oriented toward pictorial processing.

Reproduction Goal Choices

[0082] The first step in the processing of pictorial images is to choosethe desired reproduction goal. The goal chosen needs to be realizablewith the intended capture and output equipment, as well as appropriatefor the intended use of the image.

Exact and Linear Reproduction

[0083] Exact and linear reproduction are where the reproduction andoriginal are identical according to some objective, physical measurementcriteria. If absolute measurement values are identical, the reproductionis exact; if relative measurement values are identical, the reproductionis linear. Exact reproduction is rarely practical, or even appropriate,with reproductions of scenes because of the differences between thescene illumination conditions and the reproduction viewing conditions.Reproductions of hardcopy on media similar to the original may be exactreproductions, because the original and reproduction can be viewed underthe same conditions. However, in such cases exact and linearreproduction are functionally identical, since the measures on whichlinear reproduction is typically based are relative to the media white.

Appearance Reproduction

[0084] Appearance reproduction is where the reproduction and originalhave the same appearance when each is viewed under specific conditions.A linear match is an appearance match if the original and reproductionare on identical media and are viewed under the same conditions, and alinear calorimetric match is an appearance match if the white referenceand viewing conditions remain constant. Currently, the only way toproduce an appearance match under any condition is with manual,trial-and-error processing. Several appearance models have beendeveloped that allow appearance matches to be produced under conditionsthat vary in specific ways, but the accuracy of the matches varies tosome extent with the different models. Appearance models tend to be mostsuccessful in dealing with changes in illumination chromaticity.Unfortunately, many other changes are also important. In fact, onecriteria for choosing photographic dyes is to minimize changes inappearance due to changes in illumination chromaticity, as long as theobserver is adapted to the illumination. Table 1 lists a number offactors affecting the appearance of photographs. TABLE 1 FactorsAffecting Appearance Human Visual System Factors (for viewing both thescene and the reproduction) Flare in the Eye Adaptation State: To theOverall Illumination Level To the Illumination Spectral CharacteristicsSpatial Variations in Adaptation Intermediate Adaptation to MultipleConditions Factors Relating to Characteristics of the Scene or Original(as viewed by the observer, as opposed to a camera or scanner) OverallIllumination Level Illumination Spectral Characteristics Colorants Used(if known) Dynamic Range Scene Key (high- or low-) Scene Content FactorsRelating to Characteristics of the Reproduction Overall IlluminationLevel Illumination Spectral Characteristics Dynamic Range and SurroundMedia Type - Surface or Illuminant Mode Surface Reflections and VeilingGlare Media Color Synthesis Characteristics - Base Material and ColorantGamut

[0085] Preferred (Pictorial) Reproduction

[0086] In photography, the most common reproduction goal is preferredreproduction, where the aim is to produce the most pleasing imageregardless of how well it matches the original. Since there is not anappearance model that deals extensively with dynamic range changes, andsince preferred reproduction is highly dependent on dynamic range, it isdifficult to say how much of preferred reproduction is an attempt toproduce a dynamic range appearance match. If a truly comprehensiveappearance model is developed, preferred reproduction may reduce toappearance matching with a slight s-shaped tone reproduction overlay andthe corresponding saturation boost. For the time being, preferredreproduction processing frequently offers the best path to excellentphotographic quality.

[0087] A minor consideration with preferred reproduction is that thenature of the reproduction desired depends slightly on the general mediaclass. In normal viewing contexts, illuminant mode color images, such asare displayed on a monitor, tend to require somewhat less s-shape thansurface mode color images, such as prints, with equivalent dynamicranges. However, quantification of preferred reproduction for digitalsystems is just beginning, and small effects are lost in the overalluncertainty. TABLE 2 Preferred Reproduction and Appearance MatchingTable 2 lists the most common reproduction goals for variousapplications of digital photography. Default Reproduction Goals forDigital Photography Input Form -> Output Form Scene TransparencyNegative Print Transparency Preferred Linear Preferred Appearance PrintPreferred Appearance Preferred Linear

[0088] In table 2, the default reproduction goal for producing atransparency from a print, or vice versa, is to produce an appearancematch. Strictly speaking, the means for achieving this has not beendeveloped because these media have significantly different dynamicranges when viewed under typical conditions. However, a roundaboutapproach can be used to achieve the desired result. If it is assumedthat the original exhibits preferred reproduction, it is possible toundo this reproduction back to a linear space, and then implementpreferred reproduction on the new media. The result will be very closeto an appearance match. This type of processing can be done using oneLUT in an appropriate, standard RGB color space, assuming the outputdevice can render the RGB data correctly.

sRGB Color Space Processing

[0089] Device performance optimization requires that pictorialprocessing algorithms can interpret the meaning of the digital imagedata they are presented. If the processing algorithms are specific toparticular devices, there are various ways in which this information canbe communicated. Device-independent performance optimization requiresthat the meaning of the data be understandable regardless of the device.The only way to accomplish this is to establish some sort of standarddata space.

[0090] Most current color management paradigms make use of a perceptualdevice-independent color space, such as CIE XYZ or CIE L*a*b*. Themotivation for this color space type is that images are meant to beviewed, and color descriptions based on psychophysical measurements bestpredict appearance. This approach is theoretically indisputable if it ispossible to construct transforms that accurately convert image data to apsychophysical space, and if the psychophysical space accuratelydescribes appearance. Unfortunately, it is not always possible toconstruct such transforms, and the lack of a totally comprehensiveappearance model may prevent current psychophysical descriptions frompredicting appearance. The use of strict perceptual color spaces canalso result in fairly intensive processing requiring high precision,since the image data may be transformed through a non-native state.

[0091] Alternatives to perceptual color spaces are physicallystandardized, but more device native “color” spaces. Such spacesdescribe the physical meaning of the data. It may also be possible tocorrelate these physical descriptions to current appearance modeldescriptions for limited sets of viewing conditions. Obvious candidatespaces of this type are standard RGB spaces. Of those available, themost appropriate are the monitor spaces. Monitor spaces have widegamuts, most images are viewed on a monitor at some point, a great dealof manual processing is accomplished using monitor feedback, and aninternationally standardized monitor space already exists, as well ascorrelations to appearance under specific viewing conditions. Standardmonitor data, when printed on photographic media using devices withindependent channels, also tends to correlate reasonably well withStatus A densitometry. This means that photographically derivedpreferred reproduction models can be applied. It is also interesting tonote that recent work in the color appearance area is indicating thatthe use of spectrally sharpened visual response functions isadvantageous. These functions are much closer to RGB than theunsharpened visual (cone) response functions. Table 3 summarizes theadvantages and disadvantages of the two types of standard color dataspaces. TABLE 3 Advantages and Disadvantages of Perceptual and sRGBColor Spaces CIE XYZ and L*a*b* Color Spaces Advantages Excellent colorappearance reproduction if the capture is colorimetric or the colorantsused in the original are known, and the viewing conditions and mediadynamic range are appropriate. Can reproduce color using unusualcolorants as long as the viewing conditions and media dynamic range areappropriate. L*a*b* is reasonably uniform perceptually. DisadvantagesThe color reproduction accuracy advantage is lost if the capture is notcolorimetric or the colorants used in the original are not known, as isusually the case with digital cameras. Color appearance prediction maybe poor if the output media dynamic range and/or viewing conditions aresignificantly different from the original. Processing may be moreextensive and require higher precision. No model is available forpreferred reproduction. If all the gamut benefits are to be realized,the image data may need to be stored at high precision, or the raw datastored with a transform. sRCB Color Space Advantages Similar to manydevice native color spaces. Transformations to sRGB tend to be simpler,more accurate, and require less precision for storage. It is lessnecessary to save the raw data with a transform. Transformations fromsRGB to output device spaces also tend to be simpler. Since sRGB imagedata can also be described perceptually, the advantages of theperceptual color spaces can be applied. Photographic preferredreproduction models can be applied. Reasonably uniform perceptually.Relatively independent channels help with signal-to-noise issues incapture. May be similar to the spectrally sharpened tristimulus metricsto be used in future appearance models. Disadvantages Colors that areout of the monitor gamut are expressed using negative values, requiringlarger data volumes.

[0092] The standard monitor data approach provides a common ground tolink perception to the native physical behavior of devices based on RGBand CMY capture or colorants. Most devices that produce outputs ofvarying dynamic range use colorants of these types. Output devices thatuse other colorants, but have dynamic ranges of around 100:1 can also beaccommodated since the monitor data can be correlated with perceptualmetrics at the fixed dynamic range. Various forms of standard monitorRGB have been successfully used by practitioners of digital photographyfor several years. Recently, a few major corporations have formalizedthis approach by proposing a specific standard monitor color space,sRGB. This proposal paves the way for the use of a standard RGB spacefor pictorial digital image processing (PDIP).

Digital Camera Processing Pipeline

[0093] The following is one embodiment of a proposed optimized pipelinefor PDIP. It combines many aspects of my invention that are not requiredto be used together. Specific capture and output devices, quantities andcolor spaces included in this pipeline are exemplary in nature and arenot intended to limit the scope of the invention in any way.

Preliminary Measurements

[0094] The gathering of preliminary information is shown schematicallyin FIGS. 1 and 2 and is represented by boxes 100 and 101-109.

[0095] 1. Set the camera gain(s) and offset(s) as they will be setduring use, hopefully the optimum settings. Ideally, the offset shouldbe set so that a focal plane exposure of zero produces a digital levelof zero after bias and dark current subtraction. The spacing of thelevels should be chosen so that virtually all the information the cameracan capture is recorded with minimal quantization error. A rule of thumbis that the standard deviation, expressed in digital levels, of any evenfocal plane exposure, should be at least 0.5 after all fixed patternnoise removal processing. Higher standard deviations are alsoacceptable, but will require more bits for image data storage.

[0096] 2. Determine the camera fixed pattern noise characteristics, suchas dark current and pixel sensitivity non-uniformity.

[0097] 3. Measure the camera response characteristics to determine thefocal plane OECF for each channel for each illumination type ofinterest. Variations due to illumination type can usually be dealt withusing a single OECF curve shape and channel multipliers. For example, aparticular camera may have relative channel sensitivities of 0.6, 0.9,and 1 with 5500 K daylight illumination and 0.2, 0.7, and 1 with 3200 Ktungsten illumination. If channel multipliers are used, it is possibleto determine a mathematical model to predict multipliers forintermediate illumination types.

[0098] 4. Determine the camera OECFs (101) for a variety of scenes withknown spatial and spectral (or channel) radiance distributions. Use thisinformation in conjunction with the focal plane OECF measurements todevise a model that estimates flare and other image-dependentnon-linearities based on focal plane image statistics (102).

[0099] 5. Determine a matrix to transform the linearized camera datainto the standard linear RGB space, using the methods described in the(most current version of the draft or international standard derivedfrom the) new ISO work item proposal draft.

[0100] 6. Measure the linearized camera spatial frequency responses andnoise power spectra for each channel in the horizontal and verticaldirections. Determine a reasonable processing kernel size and constructa maximum information throughput spatial reconstruction kernel. Notethat each channel may require a different reconstruction kernel, andthat the ideal reconstruction kernel may vary with focal plane exposure.

[0101] 7. Measure the neutral EOCF, the spatial frequency response, andthe noise power spectrum of the output devices on which the image datamay be rendered. If an output device has minimal noise, it may not benecessary to measure the noise power spectrum. Spatial frequencyresponse and noise power spectrum measurements are also not necessarywith halftoning output devices that incorporate perceptual noisesuppression and sharpening into the halftoning algorithm. If the outputdevice is not known, the EOCF for sRGB can be used.

Processing Step 1: Determination of Flare and Scene Key

[0102] 1. Divide up the image data into the color channels, if required(202).

[0103] 2. Pixel average (boxcar filter (301) and sub-sample (302)) eachchannel to obtain scaled versions of the original image, and preferablystore the scaled images. Other blur filters can be used, such asGaussian and median. The scaled images are preferably reduced images of10,000 to 20,000 digital values. Previous work has indicated thatreduced images with a short dimension of about 100 pixels are optimalfor most subject matter and output forms. The pixel averaging is done inwhat is assumed to be an approximately logarithmic gamma type cameradata space because geometric means are preferable to arithmetic means.If the camera is linear, it would be best to convert the digital valuesto a log or gamma space prior to the pixel averaging, and then backagain, although the effect of this rather intensive additionalprocessing should be minimal.

[0104] 3. Transform the reduced image values to focal plane exposuresusing the inverse focal plane OECFs for each channel, using theappropriate gains and/or channel multipliers for the scene or originalillumination type (401).

[0105] 4. Determine the minimum,,maximum, and mean exposures, and theminimum, maximum, and mean log exposures for each channel. Estimate theimage-specific camera OECFs for each channel based on these statistics(or a subset thereof) using the previously determinedflare/non-linearity model, and the focal plane OECFs for theillumination type used (402).

[0106] 5. The scene key is determined by subtracting the average of theminimum and maximum log exposures, which is the expected mean logexposure, from the mean log exposure.

Processing Step 2: Determination of Scene Zone 1, Zone 5, and Zone 9Luminances

[0107] 1. Transform the reduced image values into linear standard RGBscene values using the processing sequence as outlined below inProcessing Steps 4 and 6 (404-407).

[0108] 2. Combine-the linear standard RGB scene values into a linearscene luminance channel using an equation appropriate for the RGB colorspace used (408). For example, if the standard color space is sRGB, theluminance conversion equation from ITU-R BT.709 is appropriate.

[0109] 3. The minimum reduced image luminance is assumed to be the sceneZone 1 luminance, the arithmetic mean luminance the Zone 5 luminance,and the maximum luminance the Zone 9 luminance (409). TABLE 4 ZoneDesignations of Perceptual Tone Categories Zone 0 - absolute black ormaximum density. Zone 1 - edge of detail in black. Zone 2 - texture inblack. Zone 3 - average dark objects. Zone 4 - dark midtones. Zone 5 -medium midtone or middle gray. Zone 6 - light midtones. Zone 7 - averagelight objects. Zone 8 - texture in white. Zone 9 - edge of detail inwhite. Zone 10 - absolute white or base white (with any fog or stain).

[0110] Processing Step 3: Determination of Output Table

[0111] 1. Select the output device and pixel pitch for the desiredrendering. If the output device is not known, assume a standard monitoras represented by sRGB.

[0112] At this point, the method proceeds along either a linearreproduction path or a preferred reproduction path, with the desiredpath depending on the particular application.

[0113] Linear Reproduction:

[0114] 2a. Determine the digital level that produces a nominal 20%reflectance, or 20% transmittance relative to the base transmittance, or20% of the white luminance, on the output device. This is designated asthe midtone reflectance level. On devices with dynamic ranges differentfrom the 100:1 total dynamic range specified for sRGB, or devices whichhave the same dynamic range but different viewing conditions, theperceptual midtone reflectance, transmittance, or luminance factor maybe different from 20%.

[0115] 3a. Determine a LUT that will produce an output with reflectances(or transmittances or luminance factors, as appropriate) that are aconstant multiplier of the scene luminances, with the constant chosen sothat the Zone 5 scene luminance reproduces at the medium midtone. Theinput to this LUT will be standard linear RGB scene values, the outputwill be standard (but not necessarily linear, depending on the outputdevice EOCF) RGB digital code values.

[0116] Preferred Reproduction:

[0117] It should be understood that the following description ofpreferred reproduction is only one way to accomplish preferredreproduction in my overall method. Many other preferred reproductionmethods can be used in the overall strategy and still fall within thescope of my invention.

[0118] 2b. Determine the maximum Zone 1 density capability of the outputmedium. This will be 90% of the maximum density capability for mostmedia. For self-luminous displays, such as monitors, “density” is equalto the negative base ten logarithm of the luminance factor of themonitor, relative to the monitor white point.

[0119] 3b Determine the viewing condition illumination level and ambientillumination level. This step may not always apply, depending on theoutput medium to be used, as well as other factors.

[0120] 4b. Determine the lesser of: the maximum Zone 1 densitycapability of the output medium, or the desired Zone 1 density based onthe viewing conditions. This will be the desired Zone 1 density.

[0121] 5b Determine the desired Zone 9 density, which is typically 0.04above the minimum density for reflection hardcopy and monitors, and 0.06above the minimum density for transparencies to be viewed on a lighttable or projected in a darkened room.

[0122] 6b. Calculate the preferred reproduction relationship between thescene luminances and output densities. The details of this calculationare as follows:

[0123] 6.1b The following quantities are needed for the calculation ofthe scene specific preferred tone reproduction curve:

[0124] Zone 1 Log Luminance—Z1logL

[0125] Zone 9 Log Luminance—Z9logL

[0126] Mean Log Exposure—{overscore (logH)}

[0127] Expected Mean Log Exposure—({overscore (logH)})

[0128] Zone 1 Output Density—Z1D

[0129] Zone 9 Output Density—Z9D

[0130] 6.2b The quantities listed in the preceding step can be used tocalculate the following important values: $\begin{matrix}{{{{Scene}\quad {Pictorial}\quad {Dynamic}\quad {Log}\quad {Range}} - {\Delta \quad \log \quad L}}{{\Delta \quad \log \quad L} = {{{Z9}\quad \log \quad L} - {{Z1}\quad \log \quad L}}}} & (1) \\{{{{Output}\quad {Pictorial}\quad {Dynamic}\quad {Log}\quad {Range}} - {\Delta \quad D}}{{\Delta \quad D} = {{Z1D} - {Z9D}}}} & (2) \\{{{{Flex}\quad {Factor}\quad {Multiplier}} - {FFM}}{{FFM} = \frac{{\Delta \quad \log \quad L} - {\Delta \quad D} + 2}{2.34}}} & (3) \\{{{{Shift}\quad {Factor}\quad {Multiplier}} - {SFM}}{{SFM} = \frac{\overset{\_}{\log \quad H} - {\langle\overset{\_}{\log \quad H}\rangle}}{0.6}}} & (4)\end{matrix}$

[0131] The preferred tone reproduction curve will be determined byadding an S-shaped flex to a reproduction curve that is linear withrespect to scene log luminance and output density. The amount of flexdepends on the scene and output dynamic ranges. The flexed curve willthen be shifted to compensate for low or high mean reflectances (asdetermined using the reduced image log exposure statistics).

[0132] 6.3b The manipulation of the base reproduction curves isaccomplished using zones. Numerical values are added to the base zonevalues to produce the desired zone values. The normalized base zonevalues, without any flex or shift, are provided in tables 5 and 6. TABLE5 Base Zone Log Luminances (BZLLs) Zone 1 Zone 2 Zone 3 Zone 4 Zone 4.5Zone 5 Zone 6 Zone 7 Zone 8 Zone 9 0 0.1613 0.3118 0.4839 0.5645 0.64520.7688 0.8763 0.957 1

[0133] TABLE 6 Base Zone Densities (BZDs) Zone 1 Zone 2 Zone 3 Zone 4Zone 4.5 Zone 5 Zone 6 Zone 7 Zone 8 Zone 9 1 0.8387 0.6882 0.51610.4355 0.3548 0.2312 0.1237 0.043 0

[0134] 6.4b The amount of flex in the desired tone reproduction curve isbased on the ratio of the scene pictorial dynamic range to the outputpictorial dynamic range. The flex is applied by adding flex zone valuesto the normalized base zone log luminances (BZLLs) listed in table 5.

[0135] 6.4.1b. The standard flex factors (SFFs), listed in table 7, arefor a standard scene pictorial dynamic range of 160:1 and a standardoutput pictorial dynamic range of 72:1. TABLE 7 Standard Flex Factors(SFFs) Zone 1 Zone 2 Zone 3 Zone 4 Zone 4.5 Zone 5 Zone 6 Zone 7 Zone 8Zone 9 0 0.087 0.086 0.036 0.004 −0.022 −0.051 −0.063 −0.048 0

[0136] 6.4.2b. The flex zone values are determined by multiplying theSFFs by the FFM.

[0137] 6.4.3b. The scene specific preferred zone log luminances aredetermined by adding the flex zone values to the BZLLs listed in table5, multiplying the sums by the ΔlogL, and adding the Z1logL (seeequation 5).

ZoneLogLuminances=Z1logL+ΔlogL(BZLL's+FFM(SFF's))  (5)

[0138] 6.5b The amount of shift in the desired tone reproduction curveis based on the difference between the mean log exposure and theexpected mean log exposure.

[0139] 6.5.1b. If the mean log exposure is lower than the expected meanlog exposure, the scene is a low key scene. The standard low key shiftfactors (SFs) are listed in table 8. TABLE 8 Standard Low Key ShiftFactors Zone 1 Zone 2 Zone 3 Zone 4 Zone 4.5 Zone 5 Zone 6 Zone 7 Zone 8Zone 9 0 0.1075 0.1344 0.1129 0.0914 0.0753 0.0538 0.0323 0.0108 0

[0140] These factors are based on a mean log exposure that is 0.6 logunits lower than the expected mean log exposure.

[0141] 6.5.2b. If the mean log exposure is higher than the expected meanlog exposure, the scene is a high key scene. The standard high key SFsare listed in table 9. TABLE 9 Standard High Key Shift Factors Zone 1Zone 2 Zone 3 Zone 4 Zone 4.5 Zone 5 Zone 6 Zone 7 Zone 8 Zone 9 00.0323 0.0484 0.0699 0.0806 0.086 0.086 0.0753 0.0484 0

[0142] These factors are based on a mean log exposure that is 0.6 logunits higher than the expected mean log exposure.

[0143] 6.5.3b. The shift zone values are determined by multiplying theappropriate standard shift factors by the SFM. The sign of the SFM isalso used to determine which set of shift factors to use. A negativesign means that the low key shift factors should be used, and a positivesign means that the high key shift factors should be used.

[0144] 6.5.4b. The scene specific preferred zone densities aredetermined by adding the shift zone values to the base zone densities(BZDs) listed in table 6, multiplying the sums by the ΔD, and adding theZ9D (see equation 6).

ZoneDensities=Z9D+ΔD (BZD's+[SFM]AppropriateSFs)  (6)

[0145] 7b. Determine a LUT that will produce preferred reproduction onthe selected output device, or a standard monitor.

[0146] Processing Step 4: Scene Linearization

[0147] 1. Subtract and divide out the fixed pattern noise (if notalready done).

[0148] 2. Construct input linearization tables by taking each possibledigital value through the image-specific inverse camera OECFs.

[0149] 3. Convert the pixel digital values to linear scene channelradiances.

[0150] Processing Step 5: Spatial Restoration

[0151] Note: This step is placed here because most spatial restorationtechniques assume the data is in a linear radiance space. Also, it isadvantageous to perform the spatial reconstruction and thetransformation to the standard linear RGB color space in the sameoperation. This processing should be placed elsewhere in the pipeline ifthe reconstruction algorithm is designed to deal with nonlinear data,either as output by the capture device, or as will be input to theoutput device.

[0152] 1. Apply the maximum information throughput (or other) spatialreconstruction kernel. As noted above, this kernel may also perform thetransformation to a standard linear RGB space (see Processing Step 6).

[0153] 2. Apply any morphological or other nonlinear processing to theimage to reduce artifacts (most common with CFA camera data).

[0154] Processing Step 6: Transformation to a standard linear RGB colorspace Note: As stated above, this step and step 5 may be donesimultaneously.

[0155] 1. Apply the linear radiance to standard linear RGB color spacetransformation matrix.

[0156] Processing Step 7: Output Processing

[0157] 1. Apply the desired output LUT. The data output by this stepwill be standard (but probably not linear) RGB digital code values. Ifthe output EOCF is the sRGB EOCF, the data will be sRGB data.

[0158] 2. Apply any subsequent output processing, such as sharpening,noise reduction, transformation of the standard RGB data to anothercolor space, application of output device-specific color LUTs,halftoning, etc.

[0159] Subsequent Processing

[0160] Image data that has been processed to a particular reproductiongoal on one output device can be processed for another output device byundoing the processing back to the point where the processed data iscommon to both output devices. Changes in reproduction goal are similar.A new reproduction goal on the same output device can be viewed as adifferent output device by the processing.

[0161] Processing Strategy for Scanning

[0162] Scanners can be viewed as digital cameras that are used to makereproductions of transparencies and prints, or as part of digital imagecapture systems that use film for the initial capture. If thereproduction goal is relative to the transparency or print, the formerviewpoint is correct, and the scanner data can be processed in the sameway as digital camera data. If the reproduction goal is relative to theoriginal scene, the latter viewpoint is more appropriate. In this caseit is necessary to consider the film camera, film, film processing, andscanner as part of the capture system. The linearization and colorcharacterization must be performed on the entire system.

[0163] Image data from scans of negatives must generally be related tothe scene, so negative film capture should be considered as part of alarger system. However, the linearization and color characterization offilm/scanner capture systems is complicated by film/process systemvariability. Also, it is important to remember that film does notcapture calorimetric information about the scene. Even if the colorantsused in the scene are known and limited in number to the number of filmcapture channels, film/process system variability typically preventsaccurate determination of calorimetric values. It is better to attemptto produce the desired reproduction directly from the film capturechannels, which are usually RGB.

[0164] With scans of negatives, it is also possible to relate areproduction goal to a conventional photographic print of the negative.This is equivalent to relating to the scene, and then using thenegative-print system tone and color reproduction characteristics as thepreferred reproduction model. However, such a model will be limited bycharacteristics of conventional photographic systems, some of which maynot be desirable and can be ignored in digital processing. Another usefor conventional photographic system tone and color reproduction modelsis to undo the preferred reproduction applied by conventionalphotographic systems so that the scanner data can be related to thescene. However, choosing to relate to the scene instead of thetransparency or print implies that the image was selected for the scenecontent, rather than for how the scene reproduced in the transparency orprint. The appropriate viewpoint must be chosen for the processingpipeline to create the desired reproduction.

[0165] In professional photography, most scenes are captured ontransparency film, and images are selected, at least to some extent,based on the appearance of the transparency, as opposed to the scenecontent. The transparency may therefore be the original to bereproduced. This reproduction is aided by the fact that, since thecolorants used in the transparency are known and limited, it is possibleto obtain calorimetrically accurate values from the scanner data. If thetransparency is to be printed on reflective media, an appearance matchmust still be created using standard RGB channels, but the colorimetricaccuracy will result in a better appearance match (to the transparency,not the scene) than if the RGB channels were not related to colormatching functions. However, if the transparency is reproduced onanother type of media, either reflective or transmissive, matching ofthe colors may not be preferred since the colors in the originaltransparency may be preferred only on the particular transparencymaterial.

[0166] In amateur photography, most scenes are captured on negativefilm, and images are selected, at least initially, based on theappearance of the scene. In some respects negatives are better suitedfor film capture than positives. They have a larger input dynamic range,more exposure latitude, and the unwanted absorptions of the film dyesare corrected using masks, making the RGB channels in a negative moreorthogonal. Also, the output dynamic range of negatives is lower thanwith transparencies, reducing flare and the dynamic range requirementsfor the scanner. Unfortunately, many scanners have difficulty producinggood image data from negatives because they are designed to scantransparencies. The level spacing is too large for the lower dynamicrange, and the processing software does not know what to do with thereversed and offset RGB color data. Another complication is thatnegative dyes are designed to modulate the red channel information atlonger wavelengths. Negatives should be scanned using channels withspectral sensitivities that allow Status M densities to be obtained, asopposed to the Status A densities appropriate for transparencies.

[0167] Print scanning is complicated by surface reflections which arehighly dependent on the scanner optical geometry. For preferredreproduction, it is best to scan the print in a way that minimizessurface reflections, such as by using the geometry specified forreflection densitometry measurements. If linear calorimetricreproduction is desired, it is necessary to simulate the viewinggeometry in the scanner. Ideally, different scanner geometries should beused for different viewing conditions. This is rarely practical,however, and the most common measurement approach for calorimetricscanner characterization is to use an integrating sphere, frequentlywith a gloss trap. However, measurements taken using this type ofgeometry will be inferior for producing preferred pictorialreproduction. Another approach is to use the geometry for reflectiondensitometry and simulate veiling glare. This approach is rare because amodel that can simulate the veiling glare characteristics of thematerial scanned is required.

[0168] User Adjustments

[0169] A comprehensive strategy for PDIP must allow for user input tothe final result. No automated routine can account for individual taste.Also, the strategy presented here does not deal with the nature ofobjects in images. These objects (such as people) can have a significanteffect on the desired reproduction. Object recognition andclassification is probably the next frontier in automated imageprocessing.

[0170] Current processing software does not provide for the quick andeasy adjustment of reproduction. This is because it is necessary to havea processing approach in place before user adjustments based on physicalmeasurements and models can be added. Automated image processing topreferred reproduction is a major advance in digital photography, butquick, easy, and intuitive tweaking of the result is almost equallyimportant. Table 10 lists a variety of user adjustment options. Theseoptions are divided into levels, so that the novice user will see onlythe simple controls, while more advanced users can choose to view moreoptions. Structures of this type are common in many softwareapplications. In all cases it is assumed that visual feedback for allchoices is provided, that user preferences can be stored, that alltechnical and device information that can be transferred automaticallyis, and that the user can request that default values be selected wherenot provided and used for all values, for some values, or the interfaceask each time. Complete descriptions of how each adjustment affects theprocessing of the image data should be available for advanced users.TABLE 10 Manual Adjustment Options Level 0 Choices: Illuminationsource - accept default or device estimation, specify type, or use aneutral balance feature. Reproduction goal - linear or preferred. Outputdevice (default to sRGB, or to a device specified in image file). Level1 Adjustments: Brightness Slider Linear reproduction - adjusts theoutput density of the scene arithmetic mean (Zone 5) luminance.Preferred reproduction (fine) - adjusts the amount of high- or low-keyshift. Preferred reproduction (coarse) - implements a gamma typebrightness shift. Contrast Slider Linear matching - disabled. Preferredreproduction (fine) - adjusts the amount of flex. Preferred reproduction(coarse) - adjusts the scaled image scale factor. Color Balance Slider(3 options) Adjust using color temperature slider. Adjust usingtri-linear coordinate (RGB) joystick. Adjust using rectangularcoofdinate (opponent color) joystick. Sharpness Slider Level 2Adjustments: Linear reproduction Choose to base the midtone on thegeometric mean. Preferred reproduction For each channel, allowadjustment of the scaled image scale factor, Zone 1 and Zone 9 outputdensity, and flare factor. Choose to base the key on the arithmeticmean. Level 3 Adjustments: Apply user specified look-up-tables andmatrices Specify look-up-table to convert each channel to sceneradiance. Specify matrix to convert to different channels for output.Specify look-up-table to produce desired reproduction on output.

[0171] The tools and techniques required for the improved, efficient,and automated processing of pictorial images are becoming available. Ifprocessing of this type is implemented, the quality of digitalphotographs should surpass that of conventional photographs in mostareas, resulting in rapid acceleration in the acceptance of digitalphotography. Parts List 100 Step of providing original image 101Sub-step of determining OECFs and OECF inverses for capture device 102Sub-step of constructing model 103 Sub-step of determining channelmultipliers 104 Sub-step of determining transformation to intermediatecolor space 105 Sub-step of determining spatial reconstruction kernels106 Sub-step of determining expected tone/color reproductioncharacteristics of output device 107 Sub-step of determining expectedviewing conditions 108 Sub-step of determining output device/mediumvisual density capabilities 109 Sub-step of selecting preferredreproduction model 200 Step of providing original image 201 Sub-step ofcapturing with capture device or reading from file 202 Sub-step ofdividing original image into channels, if required 203 Sub-step ofremoving noise, if required 300 Step of constructing scaledimage/version of original image 301 Sub-step of spatially blurringoriginal image 302 Sub-step of sub-sampling original image 400 Step ofanalyzing scaled image/version 401 Sub-step of transforming scaled imageinto focal plane data 402 Sub-step of determining significantstatistical values of focal plane data 403 Sub-step of determining flarecharacteristics or non-linear characteristics of original image 404Sub-step of determining input linearization information (LUTs orfunctions) 405 Sub-step of applying input linearization information toscaled version to produce linear scene integrated channel radiance data406 Sub-step of normalizing scaled version channel radiance data 407Sub-step of transforming normalized/balanced channel radiance data intointermediate color space to produce transformed normalized scaledversion data 408 Sub-step of combining all channels of transformednormalized scaled version data into a scene luminance channel 409Sub-step of determining significant statistical values of sceneluminance 410 Sub-step of selecting desired manner of tone reproduction411 Sub-step of calculating desired tone reproduction 412 Sub-step ofdetermining EOCF and inverse EOCF of output device 413 Sub-step ofdetermining output LUTs 500 Step of processing original image 501Sub-step of applying input linearization information to original imageto produced linearized channel data 502 Sub-step of applying spatialreconstruction kernels to linearized channel data to producereconstructed channel data 503 Sub-step of multiplying reconstructedchannel data be channel multiplier to produce normalized/balancedchannel data 504 Sub-step of transforming normalized/balance channeldata to intermediate color space to produce intermediate channel data505 Sub-step of applying output LUTs to intermediate channel data toproduce output/processed image

I claim:
 1. A method of pictorial digital image processing comprisingthe steps of: providing an original image; gathering preliminaryinformation; constructing one or more of an image-dependent flare model,an image-dependent non-linearity model, an image-dependent preferredreproduction model, and an output-dependent preferred reproduction modelin accordance with the preliminary information; and processing theoriginal image in accordance with the preliminary information and theone or more of an image-dependent flare model, an image-dependentnon-linearity model, an image-dependent preferred reproduction model,and an output-dependent preferred reproduction model to produce aprocessed image from the original image.
 2. A method of pictorialdigital image processing comprising the steps of: providing an originalimage; constructing a scaled image based on the original image; derivingimage statistics from the scaled image; and processing the originalimage in accordance with the image statistics derived from the scaledimage.
 3. The method of claim 2 wherein the step of processing theoriginal image comprises using the image statistics to determineimage-dependent linearization information and applying the linearizationinformation to one of the original image and the scaled image.
 4. Themethod of claim 2 wherein the step of processing the original imagecomprises determining image-dependent reproduction characteristics fromthe image statistics and applying the reproduction characteristics toone of the original image and the scaled image.
 5. The method of claim 2wherein the step of constructing a scaled image comprises spatiallyblurring the original image to produce a blurred image and sub-samplingthe blurred image.
 6. A method of pictorial digital image processingcomprising the steps of: determining scene luminance statistics of anoriginal image; determining output media density ranges of output mediato be used for viewing a processed image; determining image- andoutput-dependent preferred reproduction curves based on the sceneluminance statistics and the output media density ranges; and applyingthe reproduction curves to the original image to produce a processedimage.
 7. The method of claim 6 wherein the step of determining thepreferred reproduction curves comprises constructing a reproductionmodel based on the image and output dependent reproduction curves. 8.The method of claim 6 wherein the step of determining the preferredreproduction curves comprises taking into account conditions under whichthe processed image will be viewed.
 9. The method of claim 6 wherein thestep of determining the output density range comprises assuming it to bean output density range of a standard output device.
 10. A method ofpictorial digital image processing wherein images are processed in anRGB color space to produce preferred tone and color reproductioncharacteristics in a processed image, the method comprising the stepsof: determining whether original image data is in the RGB color space,which RGB color space includes a plurality of channels; transforming theoriginal image data into the RGB color space if the original image datais not already in the RGB color space; determining a tone reproductiontransformation using a preferred reproduction model; and applying thetone reproduction transformation to each of the RGB channels to producethe preferred tone reproduction characteristics, as well as theassociated color reproduction characteristics, in a processed image. 11.The method of claim 10 wherein the step of determining a tonereproduction transformation comprises determining a relationship betweena luminance channel of the original image data and a luminance channelof the processed image data.
 12. A method of pictorial digital imageprocessing wherein image data relating to an original reproduction on afirst output medium are processed for a desired reproduction on a secondoutput medium, the method comprising the steps of: specifying a natureof the original reproduction on the first output medium; obtaining imagedata related to the original reproduction on the first output medium,either data used to produce the original reproduction or data obtainedby capturing the original reproduction; transforming the image datarelated to the original reproduction to a color space appropriate forapplying a preferred reproduction model, if necessary; and processingdata for the desired reproduction on the second output medium based onthe preferred reproduction model.
 13. The method of claim 12 wherein thefirst and second output media are the same, but the original and desiredreproduction are different.
 14. The method of claim 12 wherein the firstand second output media are different, but the original and desiredreproduction are the same.
 15. The method of claim 12 wherein the firstand second output media and the original and desired reproduction aredifferent.
 16. A method of pictorial digital image processing comprisingthe steps of: providing an original image; constructing a preferredreproduction model; processing the original image in accordance with thepreferred reproduction model to produce a processed image; and allowingadjustment of the processed image by providing for adjustment of aparameter of the preferred reproduction model.
 17. The method of claim16 wherein the parameter for which adjustment is provided is one or moreof: a channel multiplier; a scale factor used to create a scaled image;a model value resulting from an estimate of a scene key made by thepreferred reproduction model; a model value resulting from an estimatemade by the preferred reproduction model of a relation between a sceneluminance range and an output density range; a scene Zone 1 outputdensity; a scene Zone 5 output density; and a scene Zone 9 outputdensity.